The PANOPTIC Camera: A Plenoptic Sensor with Real-Time Omnidirectional Capability

A new biologically-inspired vision sensor made of one hundred “eyes” is presented, which is suitable for real-time acquisition and processing of 3-D image sequences. This device, named the Panoptic camera, consists of a layered arrangement of approximately 100 classical CMOS imagers, distributed over a hemiThis research has been partly conducted with the support of the Swiss NSF under grant number 200021-125651. H. Afshari, A. Schmid and Y. Leblebici are with the Microelectronic Systems Laboratory, L. Bagnato and P. Vandergheynst are with the Signal Processing Laboratory of the Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland. L. Jacques is with the Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM), Electrical Engineering Department (ELEN), University of Louvain (UCL), Louvain-la-Neuve, Belgium. He is funded by the Belgian Science Policy (Return Grant, BELSPO) joined to the Belgian Interuniversity Attraction Pole IAP-VI BCRYPT. H. Afshari (B) · L. Jacques · L. Bagnato · A. Schmid · P. Vandergheynst · Y. Leblebici EPFL-STI-IEL-LSM, Bldg ELD, Station 11, 1015 Lausanne, Switzerland e-mail: [email protected] L. Jacques e-mail: [email protected] L. Bagnato e-mail: [email protected] A. Schmid e-mail: [email protected] P. Vandergheynst e-mail: [email protected] Y. Leblebici e-mail: [email protected] sphere of 13 cm in diameter. The Panoptic camera is a polydioptric system where all imagers have their own vision of the world, each with a distinct focal point, which is a specific feature of the Panoptic system. This enables 3-D information recording such as omnidirectional stereoscopy or depth estimation, applying specific signal processing. The algorithms dictating the image reconstruction of an omnidirectional observer located at any point inside the hemisphere are presented. A hardware architecture which has the capability of handling these algorithms, and the flexibility to support additional image processing in real time, has been developed as a two-layer system based on FPGAs. The detail of the hardware architecture, its internal blocks, the mapping of the algorithms onto the latter elements, and the device calibration procedure are presented, along with imaging results.